AutoCorrel II: a neural network event correlation approach

Intrusion detection analysts are often swamped by multitudes of alerts originating from installed intrusion detection systems (IDS) as well as logs from routers and firewalls on the networks. Properly managing these alerts and correlating them to previously seen threats is critical in the ability to effectively protect a network from attacks. Manually correlating events can be a slow tedious task prone to human error. We present a two-stage alert correlation approach involving an artificial neural network (ANN) autoassociator and a single parameter decision threshold-setting unit. By clustering closely matched alerts together, this approach would be beneficial to the analyst. In this approach, alert attributes are extracted from each alert content and used to train an autoassociator. Based on the reconstruction error determined by the autoassociator, closely matched alerts are grouped together. Whenever a new alert is received, it is automatically categorised into one of the alert clusters which identify the type of attack and its severity level as previously known by the analyst. If the attack is entirely new and there is no match to the existing clusters, this would be appropriately reflected to the analyst. There are several advantages to using an ANN based approach. First, ANNs acquire knowledge straight from the data without the need for a human expert to build sets of domain rules and facts. Second, once trained, ANNs can be very fast, accurate and have high precision for near realtime applications. Finally, while learning, ANNs perform a type of dimensionality reduction allowing a user to input large amounts of information without fearing an efficiency bottleneck. Thus, rather than storing the data in TCP Quad format (which stores only seven event attributes) and performing a multi-stage query on reduced information, the user can input all the relevant information available and instead allow the neural network to organise and reduce this knowledge in an adaptive and goal-oriented fashion. DRDC Ottawa TM 2005-193 i